On Quantifying the Accuracy of Maximum Likelihood Estimation of Participant Reliability in Social Sensing

This paper presents a confidence interval quantification of maximum likelihood estimation of participant reliability in social sensing applications. The work is motivated by the emergence of social sensing as a data collection paradigm, where humans perform the data collection tasks. A key challenge in social sensing applications lies in the uncertain nature of human measurements. Unlike well-calibrated and well-tested infrastructure sensors, humans are less reliable, and the likelihood that participants’ measurements are correct is often unknown a priori. Hence, it is hard to estimate the accuracy of conclusions made based on social sensing data. In previous work, we developed a maximum likelihood estimator of reliability of both participants and facts concluded from the data. This paper presents an analytically-founded bound that quantifies the accuracy of such maximum likelihood estimation in social sensing. A confidence interval is derived by leveraging the asymptotic normality of maximum likelihood estimation and computing the approximation of Cramer-Rao bound (CRB) for the estimation parameters. The proposed quantification approach is empirically validated and shown to accurately bound the actual estimation error given sufficient number of participants under different sensing topologies.